\documentclass[12pt,a4paper]{article}
\def\version{5.2.1}
\def\qe{{\sc Quantum ESPRESSO}}
\def\qeforge{\texttt{qe-forge.org}}
\textwidth = 17cm
\textheight = 25cm
\topmargin =-1 cm
\oddsidemargin = 0 cm

\usepackage{html}

% BEWARE: don't revert from graphicx for epsfig, because latex2html
% doesn't handle epsfig commands !!!
\usepackage{graphicx}

\def\configure{\texttt{configure}}
\def\configurac{\texttt{configure.ac}}
\def\autoconf{\texttt{autoconf}}
\def\make.sys{\texttt{make.sys}}
\def\Makefile{\texttt{Makefile}}

\def\qeImage{quantum_espresso.pdf}

\begin{document} 
\author{}
\date{}
\title{
  \includegraphics[width=5cm]{\qeImage} \\
  % title
  \Huge Developer's Manual for \\ \qe\ (v.\version) 
}
\maketitle

\tableofcontents

\newpage

\section{Introduction}

\subsection{Who should read (and who should {\em write}) this guide}

The intended audience of this guide is everybody who wants to:
\begin{itemize}
\item know how \qe\ works internally;
\item modify/customize/add/extend/improve/clean up \qe;
\item know how to read and use data produced by \qe.
\end{itemize}
The same category of people should also {\em write} this guide, of course.

\subsection{Who may read this guide but will not necessarily profit from it}

People who want to know about the capabilities of \qe,
or who want just to use it, should read the User Guide
instead of (or in addition to) this guide. In addition
to the general User Guide, there are also package-specific
guides.

People who want to know about the methods or the physics
behind \qe\ should read first the relevant  
literature (some pointers in the User Guide).

\subsection{How to contribute to \qe\ as a user}

You can contribute to a better \qe, even as an ordinary user, by:
\begin{itemize}
\item Answering other people's questions on the mailing list (correct
  answers are strongly preferred to wrong ones). 
\item Porting to new/unsupported architectures or configurations: see
  Sect. \ref{SubSec:Inst}, "Installation mechanism". You should
  not need to add new preprocessing flags, but if you do, 
  see Sect. \ref{SubSec:CPP}, "Preprocessing".
\item Pointing out bugs in the software and in the documentation
  (reports of real bugs are strongly preferred to reports of
  nonexistent bugs). See Sect. \ref{SubSec:Bugs}, "Guidelines 
  for reporting bugs".
\item Improving the documentation (generic complaints or suggestions
  that "there should be this and that" do not qualify as improvements). 
\item Suggesting changes: note however that suggestions requiring a
  significant amount of work are welcome only if accompanied by
  implementation or by a promise of future implementation (fulfilled
  promises are strongly preferred to forgotten ones). 
\item Adding new features to the code. If you like to have something
  added to \qe, contact the developers via the
  \texttt{q-e-developers[.at.]qe-forge[.dot.]org} mailing list.
  Unless there are technical reasons not to include your changes, we 
  will try to make you happy (no warranty that we will actually succeed).
\end{itemize}

\newpage

\section{\qe\ as a distribution}

\qe\ is not organized as a monolithic code, but rather as a
{\em distribution} (integrated suite) of ``packages'', with 
varying degrees of integration, that can be installed on demand,
or sometimes independently. There is a ``shell'' structure,
with at the center the {\em core} distribution, including
\begin{itemize}
\item scripts, installation tools, libraries, common source files; 
\item basic packages
\begin{itemize}
\item \texttt{PWscf}: self-consistent calculations, structural optimization,
molecular dynamics on the ground state;
\item \texttt{CP}: Car-Parrinello molecular dynamics;
\item \texttt{PostProc}: data analysis and plotting (requires \texttt{PWscf}).
\end{itemize}
\end{itemize}
Note that some libraries are downloaded on demand from the web
during the installation of the core distribution. Then comes a first
outer shell of {\em additional} packages, that can be downloaded and 
installed from the core distribution using \texttt{make}:
\begin{itemize}
\item \texttt{atomic}: pseudopotential generation
\item \texttt{PHonon}: Density-Functional Perturbation Theory
\item \texttt{NEB}: reaction pathways and energy barriers 
\item \texttt{PWCOND}: ballistic conductance
\item \texttt{XSPECTRA}: calculation of X-Ray spectra
\item \texttt{TDDFPT}: Time-dependent DFPT (requires \texttt{PHonon})
\end{itemize}
All these packages use routines from the core distribution.

A second shell of additional packages, also downloaded and installed 
on demand from the core distribution, includes
\begin{itemize}
\item \texttt{GIPAW}: calculation of NMR coefficients and chemical shifts,
\item \texttt{EPW}: electron-phonon (under development, 
requires \texttt{PHonon}).
\end{itemize}
The only difference between the ''first'' and ''second'' shell is that
the latter are stored in separate SVN repositories.

In a third shell of additional packages we find
\begin{itemize}
\item \texttt{GWL}: GW calculations using Lanczos chains.
\end{itemize}
This also uses routines from \qe, but it must be separately 
downloaded and installed.

There is then a shell of {\em external} packages, which typically
read data produced by \qe\ but do not need it to work. Some of them
(notably Yambo and WanT) can be automatically downloaded and installed
from the core distribution using \texttt{make}.

Finally there are {\em plugins}: these modify \qe\ packages, adding
new functionalities. The only plugin currently released is Plumed
(metadynamics), but other may come soon.

\section{How to become a developer}

If you want to get involved as a developer and contribute serious
or nontrivial stuff (or even simple and trivial stuff), you should
first of all register on \qeforge\ as a developer for the \qe\ 
project.

\subsection{About \qeforge}

\qeforge\ is the portal for \qe\ developers, contributors,
and for anybody else wanting to develop a project in the
field of atomistic simulations. \qeforge\ provides for
each project a repository, mailing lists, a wiki, upload 
space, a bug tracking facility, various other tools that 
are useful for developers.

Once you have obtained an account ({\em please} follow the 
instructions and introduce yourself when you register: the 
site administrator has to be sure that you are a real person!)
you may open your own project, retaining all rights on it 
(including the right not to release anything): at the end of
the "projects" page, click on the link "add new project", fill 
the form (note that the Unix name given to the project cannot
be modified). You have the choice between a repository using 
CVS, SVN, \texttt{git}, plus other choices.

You may as well register as a developer 
in an existing project: go to the project page, click on
button ''Request to become a developer'' under the ''Activity''
graph on the top of the column at the right, to obtain the permission 
from the administrator of the project.

You need to register your SSH keys in order to have read-write 
access the repository (if you have such permissions). Generate
keys on your work machine if you haven't already, using command
\texttt{ssh-keygen -t rsa}. The keys are typically found
 in file \texttt{.ssh/id-rsa.public}. Then
\begin{enumerate}
\item login to your \qeforge\ account
\item click on My stuff (menu on top line)
\item click on My account (menu on the left)
\item click on Edit SSH Keys, add your keys (be careful not to add blanks,
breaks, etc.).
\end{enumerate}

\subsection{\qe\ on \qeforge}

Currently \qe\ uses the following development tools:
\begin{itemize}
\item SVN server (with web interface to browse the repository)
\item Bug Tracking facility
\item Upload space (with download counter)
\item Mailing lists, currently
\begin{itemize}
\item \texttt{pw\_forum} (high traffic): for ordinary users, 
requests for help, problems, discussions, etc.. Only
registered users can post.
\item \texttt{q-e-developers} (low traffic):
for communications among developers and people interested
in the development of \qe. Only registered users can post
but messages from unregistered users are monitored and 
approved if relevant.
\item \texttt{q-e-commits}(medium traffic): for automatic
commit messages. Note that replies to commit messages go
to the mailing list: in case of doubts or questions or 
remarks over a specific commit, feel free to reply.
\end{itemize}
\end{itemize}
Everybody is encouraged to explore other capabilities of \qeforge.

All \qe\ developer are {\em strongly} invited to subscribe to the 
two mailing lists \texttt{q-e-developers} and \texttt{q-e-commits}.
Those who don't lose i) the opportunity to follow what is going on,
ii) the right to complain if something has gone into a direction 
they don't like. Note that subscription to mailing lists is not 
automatic when you register: you should subscribe using the links 
in \texttt{http://www.qe-forge.org/gf/project/q-e/mailman/}.
Please also consider subscribing to the bug tracker: select the
"Tracker" item on the left, then select "Bugs", then click on
"Start monitoring". You will receive an e-mail every time a
bug is filed.

\subsection{Contributing new developments}

Various procedures can be followed to contribute new developments.
It is possible to contribute:
\begin{itemize}
\item a small (or large) piece of code to an existing package; or
\item a new package that uses \qe\ as a library; or
\item a ``plugin'' that modifies \qe, adding a new functionality; or
\item a new ``external'' package that just reads data file produced by QE.
\end{itemize}
The ideal procedure depends upon the kind of project you have in
mind. In all cases, you should learn how to use SVN: see Sect.\ref{Sec:SVN}, 
"Using SVN". The three typical cases are:
\begin{itemize}
\item[a)] If your project involves changes or additions affecting only 
a small part of \qe, it is usually convenient to work directly on 
the main SVN repository (the "trunk").
\item[b)] 
If your project involves major or extensive changes to the core of
\qe, it may be a good idea to make a SVN "branch" and work on it.
Note that your branch will necessarily be public, since the SVN
trunk is public.
\item[c)]
If your project involves a major new addition (e.g. a new package),
or if you do not want it to be public during its development,
it may be a good idea to register it as a new \qeforge\ project
with a separate SVN repository. It is possible to restrict access
to selected \qe\ developers; or to keep it private; or to have
two repositories, one public and one private. It is possible to 
have the public repository automatically downloaded into the
SVN copy of \qe\ (see Sect.\ref{SubSec:propedit}).

\end{itemize}

For case a), you should from time to time update your copy (using command
\texttt{svn update}), verify if changes made meanwhile by other developers
conflict with your changes. Conflicts are in most cases easy to solve:
see Sect. \ref{SubSec:Conflicts} for hints on how to 
remove conflicts and on how to figure out what went wrong.
Once you are happy with your modified version, you can commit your
changes, or ask one of the expert developers to do this if you do not
feel confident enough.

For case b), you should from time to time align your branch with the
trunk. See Sect. \ref{SubSec:Merge} for hints on how to do this.

For case c): if your project is ``loosely coupled'' to \qe, that is,
it just uses the \qe\ installation procedure and/or data files, there
shouldn't be any major problems, since major incompatible changes are
very rare (note however that the files produced by the phonon code
change more frequently). If your project is ``tightly bound'', i.e. 
it uses routines from \qe, it is prudent to notify the other
developers.

\subsection{Hints, Caveats, Do's and Dont's for developers}

\begin{itemize}
\item Before doing anything, inquire whether it is already there,
or under development. In particular, check (and update) the "Road Map"
page \texttt{www.quantum-espresso.org/road-map}, send a message to
\texttt{q-e-developers}.
\item Before starting writing code, inquire whether you can reuse
code that is already available in the distribution. Avoid redundancy: 
the only bug-free software line is the one that doesn't exist.
\item When you make some changes:
\begin{itemize}
\item Check that are not spoiling other people's work. In particular, 
search the distribution for codes using the routine or module you are 
modifying and change its usage or its calling arguments everywhere.
Use the commit message to notify all developers if you introduce any 
``dangerous'' change (i.e. susceptible to break some features or 
packages, including external packages using \qe).
\item Do not forget that your changes must work on many different 
combinations of hardware and software, in both serial and parallel execution.
\item Do not forget that your changes must work for a wide variety of
different case: if you implement something that works only in some 
selected cases, that's ok, as long as the code stops (or at least,
issues a warning) in all other cases. There is something worse than
no results: wrong results.
\item Do not forget that your changes must work on systems of wildly
different computational size: a piece of code that works fine for
crystal silicon may gobble a disproportionate amount of time and/or
memory in a 1000-atom cell.
\end{itemize}
\item Document your contributions:
\begin{itemize}
\item If you modify what a code can do, or introduce
incompatibilities with previous versions (e.g. old data file 
no longer readable, old input no longer valid), {\em please}
report it in file \texttt{Doc/release-notes}.
\item If you add/modify/remove input variables, document
it in the appropriate
\texttt{INPUT\_*.def} file; if
you remove an input variable, update tests and examples
accordingly.
\item All newly introduced features or variables must be 
accompanied by an example or a test or both (either a 
new one or a modified existing test or example).
\end{itemize}
\item Please do not include files (any kind, including
pseudopotential files) with DOS \^{}M characters or 
tabulators \^{}I. 
\item When you modify the program sources, run the
\texttt{install/makedeps.sh}  script  or type \texttt{make depend} 
to update files \texttt{make.depend} in the various 
subdirectories. These files are under SVN as well;
if modified they should be committed.
\end{itemize}

\subsection{Guidelines for reporting bugs}

\label{SubSec:Bugs}
\begin{itemize}
\item Before deciding that a problem is due to a bug in the codes, 
verify if it is reproducible on different machines/architectures/phases
of the moon: erratic or irreproducible problems, especially in parallel
execution, are often an indication of buggy compilers or libraries
\item Bug reports should preferably be filed using the bug tracking 
facility at \qeforge:\\
\texttt{http://qe-forge.org/gf/project/q-e/tracker}
\item Bug reports should include enough information to be reproduced: 
the error message alone is seldom a sufficient piece of information. 
Typically, one should report
\begin{itemize}
\item version number, hardware/software combination(s) for which 
     the problem arises
\item whether it happens in serial or parallel execution or both 
(if in parallel only, how executed), 
\item an output for a test case showing the presumed bug
\item all the needed info and data to re-run the test case showing
the bug
\end{itemize}
The provided input should be simple and quick to execute.
\item If a bug is found in a stable (released) version of \qe, it must
be reported in the \texttt{Doc/release-notes} file.
\end{itemize}

\section{Stable releases and development cycle}
Stable releases are labelled as $N.M.p$, where $N$=major, $M$=minor, 
$p$=bugfix. The logic goes more or less as follows:
\begin{itemize}
\item {\em Major}: when something really important changes, e.g.
\begin{enumerate}
\item[v.1] First public release of PWscf
\item[v.2] Conversion from f77 to f90
\item[v.3] Merge with the CP and FPMD codes (beginning of \qe)
\item[v.4] New XML-based data file format
\item[v.5] Major package and directory reorganization
\end{enumerate}
(the above numbers are a slightly idealized versions of how things have 
gone until now)
\item {\em Minor}: when some important new functionality is being added
\item {\em Bugfix}: only bug fixes; occasionally, minor new functionalities
that don't break any existing ones are allowed to sneak into a bugfix release.
\end{itemize}
It may be convenient to make a SVN branch at release $N.M.0$: this allows 
to go on with the development while keeping track of bug fixes. 

Since release 5.2 (June 20, 2015), stable release are packaged at
fixed dates. The initial schedule is a release every three months.

Releases are stored to \qeforge. Given the size of the complete distribution,
the release is split into a ``core'' distribution and ``add-ons'',
additional packages, that can be downloaded and installed on demand 
from the core distribution. ``External'' packages can be independently
released, as long as there is no compatibility problem.

The automatic downloading of packages is implemented in file
\texttt{install/plugins\_makefile} and configured in file
\texttt{install/plugins\_list}. For independently released packages,
it is sufficient to update links. For add-ons packages, not contained
in the core distribution, there is however a catch-22: 
the core distribution must know the link to all packages it downloads,
but these are known only {\em after} such packages are uploaded to 
\qeforge\ (and the only way to discover the exact link is to go over
the released package with the mouse).
The workaround is that the core distribution looks for generic names,
written in file \texttt{install/plugins\_list}. These names are translated
by \qeforge\ into specific names. After all packages have been uploaded,
file \texttt{/var/lib/gforge/archives/index.php}, residing on \qeforge,
must be edited and links updated. This requires system privileges on
the machine hosting \qeforge.

\paragraph{Preparing a release} 
When the release date approaches, development of new stuff is temporarily
stopped: nothing new or potentially ''dangerous'' is added, and all 
attention is dedicated to fix bugs and to stabilize the distribution.
This manual and the user manual have to be updated.

Edit the script \texttt{dev-tools/release.sh} to make tarballs.

\paragraph{Updating web site} After the release has been uploaded to
\qeforge, the online documentation must be copied to directory
\texttt{/var/www/quantum\_wp\_db/wordpress-3.1.4/wp-content/uploads/Doc}
on the web site (this requires system privileges on the machine hosting
the web server).

\section{Structure of the distribution}

Since v.5, the directory structure of \qe\ reflects its organization
into packages. Each package is stored into a specific subdirectory. 
In addition, there is a set of directories, common to all packages, 
containing common code, 
libraries, installation utilities, general documentation.

Common files and directories in the \texttt{espresso/} directory are:
\begin{verbatim}
   install/                 configure
   include/                 make.sys
   archive/                 Makefile
   dev-tools/               License
   pseudo/                  README
   Doc/                     environment_variables
   clib/                    flib/
   Modules/                 upftools/
   COUPLE/
\end{verbatim}
Apart from  \texttt{License} and \texttt{README} whose meaning is 
obvious, the other files and directories are related to
\begin{itemize}
\item {\em Installation} (i.e. compilation and linking):\\
\texttt{install/}, \texttt{dev-tools/}, \texttt{archive/},
\configure, \make.sys
\item  {\em Testing} (running tests and examples):\\
\texttt{pseudo/}, \texttt{environment\_variables}
\item  {\em General documentation} (not package-specific):
\texttt{Doc/}
\item {\em  C and Fortran Libraries, modules} (F95):
\texttt{clib/}, \texttt{flib/}, \texttt{Modules/}, \texttt{COUPLE/} 
(the latter contains code and documentation useful to call \qe\ programs
 from external codes).
\end{itemize}

The core distribution also  contains the three package-specific directories
\texttt{PW/}, \texttt{PP/}, \texttt{CPV/}, for
 \texttt{PWscf}, \texttt{PostProc}, \texttt{CP}, respectively.
Typical subdirectory structure of a directory containing a package 
(e.g. \texttt{PW/}):
\begin{verbatim}
   Makefile
   examples/
   tests/
   Doc/
   src/
\end{verbatim}
Note that:
\begin{itemize}
\item \texttt{tests/} contains automated post-installation tests
(only in \texttt{PW/} and \texttt{CPV/}) while \texttt{examples/}
are not suitable for automated checks;
\item other packages may have a slightly different structure (in
particular, \texttt{PHonon} has three directories for sources
and none is called \texttt{src/} ).
\end{itemize}

\subsection{Installation Mechanism}

\label{SubSec:Inst}
Let us review the files related to compilation and linking:
\begin{itemize}
\item[--] \texttt{install/}: documentation and utilities for compilation 
and linking 
\item[--] \configure: wrapper for \texttt{install/configure} script
\item[--] \make.sys: produced by \texttt{configure}, contains 
machine-specific compilation and linking options
\item[--] \Makefile: contains dependencies and targets used by 
command \texttt{make}. 
\item[--] \texttt{include/}: files to be included into sources, to be 
pre-processed.
\end{itemize}
\texttt{./configure} {\em options} runs \texttt{install/configure},
produces file \make.sys. Its behavior can be changed by
modifying file \texttt{install/configure.ac} (see Sec.\ref{SubSec:conf}
for more details) and running (in \texttt{install/}) command \autoconf. 
This produces a new version of \texttt{install/configure}. 

\texttt{make} {\em target} checks for dependencies, recursively goes 
into subdirectories executing \texttt{make} again. The behavior of 
\texttt{make} is thus
determined by many \Makefile's in the various directories. The
most important files are \Makefile's in the directories containing
sources, e.g. \texttt{Modules/Makefile}, \texttt{PW/src/Makefile}.

Dependencies of Fortran files are contained in \texttt{make.depend} files
in each source directory. These files {\em must be updated} if you change
the sources, running script \texttt{install/makedeps.sh} or using command
\texttt{make depend}.

\paragraph{make.sys}
This file is produced by \configure\ using the template in 
\texttt{install/make.sys.in} and contains all system-specific 
information on
\begin{itemize}
\item C and Fortran compilers name, pre-processing and compilation options
\item whether the Fortran compiler performs C-style preprocessing or not
\item whether compiling for parallel or serial execution
\item available optimized mathematical libraries, libraries to be downloaded
\item Miscellanous stuff
\end{itemize}
The \make.sys\ file is included into all \Makefile's, 
using the corresponding syntax. The best documentation for the 
\make.sys\ file is the file itself. Note that if you want to
change something or to add more documentation into this file,
you may need to modify the template file \texttt{install/make.sys.in}. 

\paragraph{Makefile}
The top-level \Makefile\ contains the instructions to download,
unpack, compile and link what is required. Sample contents
(comments in italic):
\begin{verbatim}
include make.sys
\end{verbatim}
{\em Contains machine- and \qe-specific definitions}
\begin{verbatim}
default :
   @echo 'to install, type at the shell prompt:'
   ...
\end{verbatim}
{\em If no target specified, ask for one, giving a list of possibilities}
\begin{verbatim}
pw : bindir mods liblapack libblas libs libiotk libenviron
    if test -d PW ; then \
    ( cd PW ; $(MAKE) TLDEPS= all || exit 1) ; fi

\end{verbatim}
{\em Target {\tt pw}: first check the list of dependencies {\tt bindir
mods ...} etc., do what is needed; then go into {\tt PW/} and give command
{\tt make all}. Note the use of {\tt exit 1}, which is required to forward
the exit status of the sub-directory make to this makefile, since the section
in parenthesis is run in a subshell and the {\tt if / fi} block will otherwise
``hide'' its the return status and make will continue in case of errors. }
\begin{verbatim}
neb : bindir mods libs pw
    ( cd install ; $(MAKE) -f plugins_makefile $@ || exit 1 )
\end{verbatim}
{\em Target {\tt neb}: do all of the above, then go into directory
{\tt install/} where {\tt make neb} using {\tt plugins\_makefile}
as Makefile will check if NEB is there, download from the network if not,
compile and link it}
\begin{verbatim}
libblas : touch-dummy
     cd install ; $(MAKE) -f extlibs_makefile $@
\end{verbatim}
{\em Target {\tt libblas}: this is an external library, that may or may
not be needed, depending upon what is written in {\tt make.sys}. If
needed, go into directory {\tt install/} where {\tt make libblas} using
{\tt extlibs\_makefile} as Makefile will check if BLAS are there, download
from the network if not,
compile and build the library}

\paragraph{PW/Makefile}
Second-level \Makefile\ contains only targets related to a given
subdirectory or package. Sample contents:
\begin{verbatim}
sinclude ../make.sys
default : all
all: pw pwtools
pw:     
    ( cd src ; $(MAKE) all || exit 1 )

pwtools: pw
    ( cd tools ; $(MAKE) all || exit 1 )

...
\end{verbatim}
{\em Target {\tt pw}: go into {\tt src/} if it exists, and (apart 
from \texttt{make} wizardry) give command {\tt make pw}. It is important
to note that {\tt pwtools} has to depend on {\tt pw} or else this
makefile will break when calling parallel make using {\tt make -j\# }
Other targets are quite similar: go into a subdirectory, e.g.
{\tt Doc/} and '{\tt make} something', e.g. {\tt make clean}.}

\paragraph{PW/src/Makefile}
The most important and most complex Makefile is the one in the
source directory. It is also the one you need to modify if you 
add something.
\begin{verbatim}
include ../../make.sys
\end{verbatim}
{\em Contains machine- and \qe-specific definitions}
\begin{verbatim}
MODFLAGS= $(MOD_FLAG)../../iotk/src
          $(MOD_FLAG)../../Modules $(MOD_FLAG).
\end{verbatim}
{\em Location of needed modules; {\rm \texttt{MOD\_FLAG}} is defined in
\texttt{make.sys}}
\begin{verbatim}
PWOBJS = \
pwscf.o
\end{verbatim}
{\em Object file containing main program (this is actually redundant)}
\begin{verbatim}
PWLIBS = \
a2fmod.o \
...
wannier_enrg.o
\end{verbatim}
{\em List of objects - add here new objects, or delete from this list. Do not
forget the backslash! It ensure continuation of the line}
\begin{verbatim}
QEMODS=../../Modules/libqemod.a
\end{verbatim}
{\em Objects from {\rm\texttt{Modules/}} are available from the above archive.
The directory where F95 modules are must also be specified to the compiler!}
\begin{verbatim}
TLDEPS=bindir mods libs liblapack libblas libenviron
\end{verbatim}
{\em TLDEPS=Top-Level DEPendencieS: a machinery to ensure proper
compilation with correct dependencies also if compiling from inside
a package directory and not from top level}
\begin{verbatim}
LIBOBJS = ../../flib/ptools.a ../../flib/flib.a
          ../../clib/clib.a   ../../iotk/src/libiotk.a
\end{verbatim}
{\em All needed QE-specific libraries}
\begin{verbatim}
all : tldeps pw.x generate_vdW_kernel_table.x
\end{verbatim}
{\em Targets that will be build - add here new executables}
\begin{verbatim}
pw.x : $(PWOBJS) libpw.a $(LIBOBJS) $(QEMODS)
     $(LD) $(LDFLAGS) -o $@ \
     $(PWOBJS) libpw.a $(QEMODS) $(LIBOBJS) $(LIBS)
   - ( cd ../../bin; ln -fs ../PW/src/$@ . )
\end{verbatim}
{\em Target {\tt pw.x} - produces executable with the same name.
It also produces a link to the executable in {\tt espresso/bin/}.
Do not forget tabulators even if you do not see them!
All variables (introduced by \$) are either defined locally
in {\tt Makefile} or imported from {\tt make.sys}}
\begin{verbatim}
libpw.a : $(PWLIBS)
        $(AR) $(ARFLAGS) $@ $?
        $(RANLIB) $@
\end{verbatim}
{\em This builds the library libpw.a - again, do not forget tabulators}
\begin{verbatim}
tldeps:
       test -n "$(TLDEPS)" && ( cd ../.. ;
       $(MAKE) $(TLDEPS) || exit 1) || :
\end{verbatim}
{\em second part of the TLDEPS machinery}
\begin{verbatim}
clean :
    - /bin/rm -f *.x *.o *.a *~ *.F90 *.d *.mod *.i *.L
\end{verbatim}
{\em There should always be a ''clean'' target, removing all compiled (*.o)
or preprocessed (*.F90) stuff - compiled F95 modules may have different
filenames: the four last items cover most cases}
\begin{verbatim}
include make.depend
\end{verbatim}
{\em Contains dependencies of objects upon other objects. Sample
content of file {\tt make.depend} (can be produced by {\tt install/makedep.sh}):}
\begin{verbatim}
a2fmod.o : ../../Modules/io_global.o
a2fmod.o : ../../Modules/ions_base.o
a2fmod.o : ../../Modules/kind.o
a2fmod.o : pwcom.o
a2fmod.o : start_k.o
a2fmod.o : symm_base.o
\end{verbatim}
{\em tells us that the listed objects must have been compiled
prior to compilation of a2fmod.o - {\tt make} will take care of this.}

{\bf BEWARE:} the Makefile system is in a stable but delicate equilibrium,
resulting from many years of experiments on many different machines.
Handle with care: what works for you may break other cases.

\subsubsection{Preprocessing}

\label{SubSec:CPP}
Fortran-95 source code contains preprocessing option with 
the same syntax used by the C preprocessor \texttt{cpp}.
Most F95 compilers understand preprocessing options \texttt{-D ...}
or some similar form. Some compilers however do not support
or do not implement properly preprocessing. In this case the
preprocessing is done using \texttt{cpp}. 
Normally, \configure\ takes care of this, by selecting the
appropriate rule \texttt{@f90rule@} below, in this section
of file \texttt{make.sys.in}:
\begin{verbatim}
.f90.o:
	@f90rule@
\end{verbatim}
and producing the appropriate file \make.sys.

Preprocessing is useful to
\begin{itemize}
\item account for machine dependency in a unified source tree
\item distinguish between parallel and serial execution when they
follow different paths (i.e. there is a substantial difference between
serial execution and parallel execution on a single processor)
\item  introduce experimental or special-purpose stuff
\end{itemize}
Use with care and {\em only when needed}. See file 
\texttt{include/defs.README} for a list of preprocessing 
options. Please {\em keep that list updated}.

{\em Note:} \texttt{include/f\_defs.h} is obsolete and
must not be used any longer.

The following capabilities of the C preprocessor are used:
\begin{itemize}
\item assign a value to a given expression. For instance, command
  \texttt{\#define THIS that}, or the option in the command line:
  \texttt{-DTHIS=that}, will replace all occurrences of \texttt{THIS}
  with \texttt{that}. 
\item include file (command \texttt{\#include})
\item expand macros (command \texttt{\#define})
\item execute conditional expressions such as
\begin{verbatim}
  #ifdef __expression
    ...code A...
  #else
    ...code B...
  #endif
\end{verbatim}
If \texttt{\_\_expression} is defined (with a \texttt{\#define} command
or from the command line with option \texttt{-D\_\_expression}), 
then  \texttt{...code A...} is sent to output; otherwise 
\texttt{...code B...} is sent to output.

\end{itemize}
In order to make  preprocessing options
easy to see, preprocessing variables should start with  
two underscores, as \texttt{\_\_expression} in the above
example. Traditionally ''preprocessed'' variables are also written in
uppercase. 

\subsubsection{How to edit the \configure\ script}

\label{SubSec:conf}
The \configure\ script is generated from its source file
\configurac\ by the GNU \autoconf\ utility
(\texttt{http://www.gnu.org/software/autoconf/}).  Don't edit \configure\
directly: whenever it gets regenerated, your changes will be lost.
Instead, go to the \texttt{install/} directory, edit \configurac, 
then run \autoconf\ to regenerate \configure. If you want 
to keep the old \configure, make a copy
first.

GNU \autoconf\ is installed by default on most Unix/Linux systems.  If
you don't have it on your system, you'll have to install it. You will
need a recent version (e.g. v.2.65) of \autoconf, because our 
\configurac\
file uses recent syntax.

\configurac\ is a regular Bourne shell script (i.e., "sh" -- not csh!), 
except that:
\begin{itemize}
\item[--] capitalized names starting with "AC\_" are \autoconf\
  macros.  Normally you shouldn't have to touch them. 
\item[--] square brackets are normally removed by the macro processor.
  If you need a square bracket (that should be very rare), you'll have
  to write two. 
\end{itemize}

You may refer to the GNU \autoconf\ Manual for more info.

\texttt{make.sys.in} is the source file for \make.sys, that
\configure\ generates: you might want to edit that file as well. 
The generation procedure is as follows: if \configurac\ contains the macro
"AC\_SUBST(name)", then every occurrence of "@name@" in the source
file will be substituted with the value of the shell variable "name"
at the point where AC\_SUBST was called.

Similarly, \configure\texttt{.msg} is generated from \configure\texttt{.msg.in}: this
file is only used by \configure\ to print its final report, and isn't
needed for the compilation.  We did it this way so that our
\configure\ may also be used by other projects, just by replacing the
\qe-specific \configure\texttt{.msg.in} by your own.

\configure\ writes a detailed log of its operation to \texttt{config.log}.
When any configuration step fails, you may look there for the relevant
error messages.  Note that it is normal for some checks to fail.

\subsubsection{How to add support for a new architecture}

In order to support a previously unsupported architecture, first you
have to figure out which compilers, compilation flags, libraries
etc. should be used on that architecture.
In other words, you have to write a \make.sys\ that works: you may use
the manual configuration procedure for that (see the 
User Guide).  Then, you have to modify \configure\ so that it can
generate that \make.sys\ automatically.

To do that, you have to add the case for your architecture in several
places throughout \configurac:
\begin{enumerate}
\item Detect architecture

Look for these lines:
\begin{verbatim}
  if test "$arch" = ""
  then
          case $host in
                  ia64-*-linux-gnu )      arch=ia64   ;;
                  x86_64-*-linux-gnu )    arch=x86_64 ;;
                  *-pc-linux-gnu )        arch=ia32   ;;
                  etc.
\end{verbatim}
Here you must add an entry corresponding to your architecture and
operating system.  Run \texttt{config.guess} to obtain the string identifying
your system.
For instance on a PC it may be "i686-pc-linux-gnu", while on IBM SP4
"powerpc-ibm-aix5.1.0.0".  It is convenient to put some asterisks to
account for small variations of the string for different machines of
the same family.  For instance, it could be "aix4.3" instead of
"aix5.1", or "athlon" instead of "i686"...

\item  Select compilers

Look for these lines:

\begin{verbatim}
  # candidate compilers and flags based on architecture
  case $arch in
  ia64 | x86_64 )
        ...
  ia32 )
        ...
  aix )
        ...
  etc.
\end{verbatim}

Add an entry for your value of \$arch, and set there the appropriate
values for several variables, if needed (all variables are assigned
some reasonable default value, defined before the "case" block):

- "try\_f90" should contain the list of candidate Fortran 90 compilers,
in order of decreasing preference (i.e. configure will use the first
it finds).  If your system has parallel compilers, you should list
them in "try\_mpif90".

- "try\_ar", "try\_arflags": for these, the values "ar" and "ruv" should
be always fine, unless some special flag is required (e.g., -X64
With sp4).  

- you should define "try\_dflags" if there is
any
"\#ifdef" specific to your machine: for instance, on IBM machines,
"try\_dflags=-D\_\_AIX" . A list of such flags can be found in file 
\texttt{include/defs.h.README}.

You shouldn't need to define the following:
- "try\_iflags" should be set to the appropriate "-I" option(s)
needed by the preprocessor or by the compiler to locate *.h files
to be included; try\_iflags="-I../include" should be good for most cases

For example, here's the entry for IBM machines running AIX:
\begin{verbatim}
   aix )
        try_mpif90="mpxlf90_r mpxlf90"
        try_f90="xlf90_r xlf90 $try_f90"
        try_arflags="-X64 ruv"
        try_arflags_dynamic="-X64 ruv"
        try_dflags="-D__AIX -D__XLF"
        ;;
\end{verbatim}
The following step is to look for both serial and parallel fortran
compilers:
\begin{verbatim}
  # check serial Fortran 90 compiler...
  ...
  AC_PROG_F77($f90)
  ...
        # check parallel Fortran 90 compiler
  ...
        AC_PROG_F77($mpif90)
  ...
  echo setting F90... $f90
  echo setting MPIF90... $mpif90
\end{verbatim}
A few compilers require some extra work here: for instance, if the
Intel Fortran compiler was selected, you need to know which version
because different versions need different flags.

At the end of the test,
 
- \$mpif90 is the parallel compiler, if any; if no parallel compiler
  is found or if \texttt{--disable-parallel} was specified, \$mpif90
  is the serial compiler 

- \$f90 is the serial compiler

Next step: the choice of (serial) C and Fortran 77 compilers.
Look for these lines:
\begin{verbatim}
  # candidate C and f77 compilers good for all cases
  try_cc="cc gcc"
  try_f77="$f90"

  case "$arch:$f90" in
  *:f90 )
        ....
  etc.
\end{verbatim}
Here you have to add an entry for your architecture, and since the 
correct choice of C and f77 compilers may depend on the fortran-90 
compiler, you may need to specify the f90 compiler as well.
Again, specify the compilers in try\_cc and try\_f77 in order of 
decreasing preference.  At the end of the test, 

- \$cc is the C compiler

- \$f77 is the Fortran 77 compiler, used to compile *.f files
(may coincide with \$f90)

\item Specify compilation flags.

Look for these lines:
\begin{verbatim}
  # check Fortran compiler flags
  ...
  case "$arch:$f90" in
  ia64:ifort* | x86_64:ifort* )
        ...
  ia64:ifc* )
        ...
  etc.
\end{verbatim}
Add an entry for your case and define:

- "try\_fflags": flags for Fortran 77 compiler.

- "try\_f90flags": flags for Fortran 90 compiler.
In most cases they will be the same as in Fortran 77 plus some
others.  In that case, define them as "\$(FFLAGS) -something\_else".

- "try\_fflags\_noopt": flags for Fortran 77 with all optimizations
turned off: this is usually "-O0".
These flags must be used for compiling flib/dlamch.f (part of our
version of Lapack): it won't work properly with optimization.

- "try\_ldflags": flags for the linking phase (not including the list
of libraries: this is decided later). 

- "try\_ldflags\_static": additional flags to select static compilation
(i.e., don't use shared libraries).

- "try\_dflags": must be defined if there is in the code any \#ifdef
specific to your compiler (for instance, -D\_\_INTEL for Intel
compilers).  Define it as "\$try\_dflags -D..." so that pre-existing
flags, if any, are preserved.

- if the Fortran 90 compiler is not able to invoke the C preprocessor
automatically before compiling, set "have\_cpp=0" (the opposite case
is the default). The appropriate compilation rules will be generated 
accordingly. If the compiler requires that any flags be specified in
order to invoke the preprocessor (for example, "-fpp " -- note the 
space), specify them in "pre\_fdflags".

For example, here's the entry for ifort on Linux PC:
\begin{verbatim}
  ia32:ifort* )
          try_fflags="-O2 -tpp6 -assume byterecl"
          try_f90flags="\$(FFLAGS) -nomodule"
          try_fflags_noopt="-O0 -assume byterecl"
          try_ldflags=""
          try_ldflags_static="-static"
          try_dflags="$try_dflags -D__INTEL"
          pre_fdflags="-fpp "
          ;;
\end{verbatim}
Next step: flags for the C compiler. Look for these lines:
\begin{verbatim}
  case "$arch:$cc" in
  *:icc )
        ...
  *:pgcc )
        ...
  etc.
\end{verbatim}
Add an entry for your case and define:

- "try\_cflags": flags for C compiler.

- "c\_ldflags": flags for linking, when using the C compiler as linker.
This is needed to check for libraries written in C, such as FFTW.

- if you need a different preprocessor from the standard one (\$CC -E),
define it in "try\_cpp".

For example for XLC on AIX:
\begin{verbatim}
  aix:mpcc* | aix:xlc* | aix:cc )
          try_cflags="-q64 -O2"
          c_ldflags="-q64"
          ;;
\end{verbatim}
Finally, if you have to use a nonstandard preprocessor, look for these
lines:
\begin{verbatim}
  echo $ECHO_N "setting CPPFLAGS... $ECHO_C"
  case $cpp in
        cpp) try_cppflags="-P -traditional" ;;
        fpp) try_cppflags="-P"              ;;
        ...
\end{verbatim}
and set "try\_cppflags" as appropriate.

\item Search for libraries

To instruct \configure\ to search for libraries, you must tell it two
things: the names of libraries it should search for, and where it
should search.

The following libraries are searched for:

- BLAS or equivalent. 
Some vendor replacements for BLAS that are supported by \qe\ are:
\begin{quote}
    MKL on Linux, 32- and 64-bit Intel CPUs\\
    ACML on Linux, 64-bit AMD CPUs\\
    essl on AIX\\
    SCSL on sgi altix\\
    SUNperf on sparc
\end{quote}
Moreover, ATLAS is used over BLAS if available.

- LAPACK or equivalent. Some vendor replacements for LAPACK that are supported by \qe\ are:
\begin{quote}
    mkl on linux
    SUNperf on sparc
\end{quote}

- FFTW (version 3) or another supported FFT library. The latter include:
\begin{quote}
    essl on aix
    ACML on Linux, 64-bit AMD CPUs
    SUNperf on sparc
\end{quote}

- the MASS vector math library on aix

- an MPI library. This is often automatically linked by the compiler

If you have another replacement for the above libraries, you'll have
to insert a new entry in the appropriate place.

This is unfortunately a little bit too complex to explain.
Basic info: \\
"AC\_SEARCH\_LIBS(function, name, ...)" looks for symbol
"function" in library "libname.a".  If that is found, "-lname" is
appended to the LIBS environment variable (initially empty).
The real thing is more complicated than just that because the
"-Ldirectory" option must be added to search in a nonstandard
directory, and because a given library may require other libraries as
prerequisites (for example, Lapack requires BLAS).
\end{enumerate}

\subsection{Libraries}

Subdirectory \texttt{flib/} contains libraries written in fortran77 
(\texttt{*.f}) and in fortran-90 (\texttt{*.f90}).
The latter should not depend on any module, except for modules
\texttt{kinds} and \texttt{constants}.

Subdirectory \texttt{clib/} contains libraries written in C 
(\texttt{*.c}). There are currently two different ways to 
ensure that fortran can call C routines. The new and recommanded 
way uses the fortran-95 intrinsic \texttt{iso\_c\_binding} module.
See \texttt{Modules/wrappers.f90} for inspiration and examples.
Reference documentation can be found for instance here:\\
{\tt 
https://gcc.gnu.org/onlinedocs/gfortran/Interoperable-Subroutines-and-Functions.html}

The old way uses macros in C routines:
\begin{enumerate}
\item \texttt{F77\_FUNC (func,FUNC)} for function \texttt{func}, not
  containing underscore(s) in name  
\item \texttt{F77\_FUNC\_(f\_nc,F\_NC)} for function \texttt{f\_nc},
  containing underscore(s) in name 
\end{enumerate}
These macros are defined in file \texttt{include/c\_defs.h},
included by all \texttt{*.c} files, and are automagically 
generated by \configure. The goal of these macros is to
choose the correct case (lowercase or uppercase, the latter
probably obsolete) and the correct number of underscores. 
See file \texttt{include/defs.h.README} for more info.

% \subsection{Adding new directories or routines}

\section{Algorithms}
% \subsection{Diagonalization}
% \subsection{Self-consistency}
% \subsection{Structural optimization}
% \subsection{Symmetrization}
\subsection{Gamma tricks}

In calculations using only the $\Gamma$ point (k=0),
the Kohn-Sham orbitals can be chosen to be real functions in 
real space, so that 
$
  \psi(G) = \psi^*(-G).
$
This allows us to store only half of the Fourier components.
Moreover, two real FFTs can be performed as a single complex FFT.
The auxiliary complex function $\Phi$ is introduced:
$
    \Phi(r) = \psi_j(r)+ i \psi_{j+1}(r)
$
whose Fourier transform $\Phi(G)$ yields

$
   \psi_j    (G) =  {\Phi(G) + \Phi^*(-G)\over 2},
   \psi_{j+1}(G) =  {\Phi(G) - \Phi^*(-G)\over 2i}.
$

A side effect on parallelization is that $G$ and $-G$ must
reside on the same processor. As a consequence, pairs of columns
with $G_{n'_1,n'_2,n'_3}$ and $G_{-n'_1,-n'_2,n'_3}$
(with the exception of the case $n'_1=n'_2=0$),
must be assigned to the same processor.

\subsection{Restart}

The two main packages, \texttt{PWscf} and \texttt{CP}, support
restarting from interrupted calculations, Restarting is trivial
in \texttt{CP}: it is sufficient to save from time to time a 
restart file containing wavefunctions, orthogonality matrix, 
forces, atomic positions, at the current and previous time step.

Restarting is much more complicated in  \texttt{PWscf}. Since v.5.1.
restarting from interrupted calculations is possible ONLY if the code
has been explicitly stopped by user. It is not practical to try to
restart from any possible case, such as e.g. crashes. This would 
imply saving lots of data all the time. With modern machines, this is
not a good idea. Restart in  \texttt{PWscf} currently works as follows:
\begin{itemize}
\item Each loop calls \texttt{check\_stop\_now} just before the end. 
  If a user request to stop is found, create a small file
  \texttt{restart\_*}, containing only loop-specific local variables; 
  close and save files used by the loop if any; set variable
  \texttt{conv\_elec} to false; return
\item After each routine containing a loop has been called, check if the code
  was either stopped there or no convergence was achieved; if so, save 
  data (if needed) for the current loop as well, return.
\item Return after return, exit all loops and go to main program, which must save
  needed global variables to file. The only difference with normal exit is that
  temporary files are kept, while files in portable format are not saved.
\item if variable \texttt{restart} is set in input:
  \begin{itemize}
  \item starting potential and wavefunctions are read from file
  \item each routine containing a loop checks for the existence of a 
    \texttt{restart\_*} file before starting its loop
  \end{itemize}
\end{itemize}
As of April 2013 only the electronic loop is organized ths way. Loops
on nuclear positions will be organized in the same manner once their
re-organization is completed. To be done:
\begin{itemize}
\item wg and et should be read from data file
\item rho(+paw/U/metagga info) should be written to and read from
  unformatted data file similar to the file used in \texttt{mix\_rho}; 
  portable format should be written only at convergence.
\end{itemize}


%\section{Structure of the code}
% \subsection{Modules and global variables}
% \subsection{Meaning of the most important variables}
% \subsection{Conventions for indices}

% \subsection{Performance issues}
% \subsection{Portability issues}

\section{Format of arrays containing charge density, potential, etc.}
The index of arrays used to store functions defined on 3D meshes is
actually a shorthand for three indices, following the FORTRAN convention
("leftmost index runs faster"). An example will explain this better.
Suppose you have a 3D array \texttt{psi(nr1x,nr2x,nr3x)}. FORTRAN
compilers store this array sequentially  in the computer RAM in the following way:
\begin{verbatim}
        psi(   1,   1,   1)
        psi(   2,   1,   1)
        ...
        psi(nr1x,   1,   1)
        psi(   1,   2,   1)
        psi(   2,   2,   1)
        ...
        psi(nr1x,   2,   1)
        ...
        ...
        psi(nr1x,nr2x,   1)
        ...
        psi(nr1x,nr2x,nr3x)
etc
\end{verbatim}
Let \texttt{ind} be the position of the \texttt{(i,j,k)} element in the above list:
the following relation
\begin{verbatim}
        ind = i + (j - 1) * nr1x + (k - 1) *  nr2x * nr1x
\end{verbatim}
holds. This should clarify the relation between 1D and 3D indexing. In real
space, the \texttt{(i,j,k)} point of the FFT grid with dimensions
\texttt{nr1} ($\le$\texttt{nr1x}),
\texttt{nr2}  ($\le$\texttt{nr2x}), , \texttt{nr3} ($\le$\texttt{nr3x}), is
$$
r_{ijk}=\frac{i-1}{nr1} \tau_1  +  \frac{j-1}{nr2} \tau_2 +
\frac{k-1}{nr3} \tau_3
$$
where the $\tau_i$ are the basis vectors of the Bravais lattice.
The latter are stored row-wise in the \texttt{at} array:
$\tau_1 = $ \texttt{at(:, 1)},
$\tau_2 = $ \texttt{at(:, 2)},
$\tau_3 = $ \texttt{at(:, 3)}.

The distinction between the dimensions of the FFT grid,
\texttt{(nr1,nr2,nr3)} and the physical dimensions of the array,
\texttt{(nr1x,nr2x,nr3x)} is done only because it is computationally
convenient in some cases that the two sets are not the same.
In particular, it is often convenient to have \texttt{nrx1}=\texttt{nr1}+1
to reduce memory conflicts.

\section{Parallelization (MPI)}

In MPI parallelization, a number of independent processes are started
on as many processors, communicating via calls to MPI libraries
(the code will work even with more than one process per processor, 
but this is not a smart thing to do). Each process has its own 
set of variables and knows nothing about other processes' variables. 
Variables that take little memory are replicated on all processors, 
those that take a lot of memory (wavefunctions, G-vectors, R-space grid) 
are distributed. 

\subsection{General rules}

Calls to MPI libraries should be confined to a few selected places,
not scattered everywhere into the source code. The vast majority
of parallel operations consist either in broadcasts from one processor
to all others, or in global operations: parallel sums and transpose.
All you need is the MPI communicator (plus the ID of the root processor
for broadcasts), and the appropriate call to wrapper routines, contained 
in {\tt espresso/Modules/mp.f90} and {\tt espresso/Modules/mp\_base.f90}. 
For instance: {\tt mp\_sum} is a wrapper to {\tt mpi\_reduce},
{\tt mp\_bcast} to {\tt mpi\_bcast}.

For efficiency reasons (latency is very significant), performing many
parallel operations on a small amount of data each must be avoided. 
If you can, store a sizable amount of data and transmit it in a single
MPI call. An example of REALLY BAD code:
\begin{verbatim}
   COMPLEX, ALLOCATABLE :: wfc(:,:), swfc(:,:)
   ALLOCATE (wfc(npwx,m),swfc(npwx,m))
   DO i=1,m
      DO j=1,m
         ps = zdotc(npw,wfc(1,i),1,swfc(1,j)1)
         CALL mp_sum(ps,intra_bgrp_group)
      END DO
   END DO
\end{verbatim}
MUCH better code, both for serial and parallel speed:
\begin{verbatim}
   COMPLEX, ALLOCATABLE :: ps(:,:), wfc(:,:), swfc(:,:)
   ALLOCATE (ps(m,m), wfc(npwx,m),swfc(npwx,m))
   CALL zgemm ('c', 'n', m, m, npw, (1.d0, 0.d0), wfc, &
               npwx, swfc, npwx, (0.d0, 0.d0), ps, m)
   CALL mp_sum(ps,intra_bgrp_group)
\end{verbatim}

\subsubsection{Usage of \#ifdef \_\_MPI}

Calls to MPI libraries require variables contained into a 
\texttt{mpif.h} file that is usually absent on serial machines.
In order to prevent compilation problems on serial machines,
the following rules {\em must} be followed:
\begin{itemize}
\item Direct calls to MPI library routines must be replaced by
calls to wrapper routines like those in module \texttt{mp.f90}. 
If this is not possible or not convenient, use \#ifdef \_\_MPI
to prevent compilation and usage in the serial case. Note that 
some compilers do not like empty files or modules containing nothing!
\item Wrapper routines do not need to be conditionally called:
there are \#ifdef's inside them. Keep the difference between serial
and parallel code to a minimum: more \#ifdef \_\_MPI may be needed
only when the flux of parallel and serial execution differ.
\item Unneeded \#ifdef \_\_MPI may be removed if already present;
\#ifdef \_\_PARA is obsolete and must not be used in new developments.
\end{itemize}

\subsection{Parallelization levels and communicators}

\texttt{mp\_world.f90} is the module containing all processors 
on which QE is running. \texttt{world\_comm} 
is the communicator between all such processors. In QE, its usage
should be confined to parallel environment initialization. It
should not be used in source code, unless this is used only by
stand-alone executables that perform simple auxiliary tasks 
and do not allow for  multiple parallelization levels.
Unless QE is started from an external code, \texttt{world\_comm}
will in practice coincides with MPI\_WORLD\_COMM.

\texttt{mp\_image.f90} is the module containing information about 
``image" parallelization, i.e. division into quasi-independent similar
calculations, each taking care of a different set of atomic
positions (NEB, PWscf) or of different irreps/phonon wavevectors
(PHonon). \texttt{intra\_image\_comm} is the communicator between 
processors of the same image (most of the action will happen here); 
\texttt{inter\_image\_comm} is the communicator between processors 
belonging to different images (should be used only when communication 
between images is necessary). 
{\tt intra\_image\_comm} and {\tt world\_comm} coincide if there 
is just one image running.

\texttt{mp\_pools.f90} is the module containing information about k-point 
(``pool") parallelization. \texttt{intra\_pool\_comm} is the communicator 
between processors working on the same group (``pool") of k-points;
\texttt{inter\_pool\_comm} is the communicator between different 
k-point pools. Note that:
\begin{quote}
\framebox{$\sum_{\bf k}\equiv$ sum over local {\bf k}-points + 
{\tt mp\_sum} on {\tt inter \_pool\_comm}}
\end{quote}
{\tt intra\_pool\_comm} and {\tt intra\_image\_comm} coincide if there 
is just one k-point pool.

\texttt{mp\_bands.f90} is the module containing information about band
parallelization. \texttt{intra\_bgrp\_comm} is the communicator between
processors of the same group of bands; \texttt{inter\_band\_comm} is the 
communicator between processors belonging to different groups of bands.
Note that band parallelization is currently implemented only in CP and
for hybrid functionals in PW. When a sum over all bands is needed:
\begin{quote}
\framebox{$\sum_i\equiv$ sum over local bands + {\tt mp\_sum} on 
{\tt inter\_bgrp\_comm}}
\end{quote}
{\tt intra\_bgrp\_comm} and {\tt intra\_pool\_comm} coincide
if there is just one band group.

Plane waves (${\bf k}+{\bf G}$ or ${\bf G}$ vectors up to the specified 
kinetic energy cutoff) are distributed across processors of the
{\tt intra\_bgrp\_comm} communicators. Sums over all plane waves
or {\bf G}-vectors (as e.g. in scalar products $\langle\phi_i|\phi_j\rangle$) 
should be performed as follows:
\begin{quote}
\framebox{$\sum_{\bf G} \equiv$ {\tt mp\_sum} on {\tt intra\_bgrp\_comm}}
\end{quote}
The same holds for real-space FFT's grid.

\subsection{Tricks and pitfalls}

\begin{itemize}
\item
Replicated calculations may either be performed independently on 
each processor, or performed on one processor and broadcast to all
others. The first approach requires less programming, but it is unsafe:
in principle all processors should yield exactly the same results, if 
they work on the same data, but sometimes they don't (depending on the
machine, compiler, and libraries). Even a tiny difference in the last 
significant digit can eventually cause serious trouble if allowed to
 build up, especially when a replicated check is performed (in which
case the code may ''hang'' if the check yields different results on 
different processors). Never assume that the value of a variable produced 
by replicated calculations is exactly the same on all processors: when in 
doubt, broadcast the value calculated on a specific processor (the ''root'' 
processor) to all others.
\item
Routine \texttt{errore} should be called in parallel by all processors,
or else it will hang
\item
I/O operations: file opening, closing, and so on, are as a rule performed 
only on processor \texttt{ionode}. The correct way to check for errors is 
the following:
\begin{verbatim}
IF ( ionode ) THEN
   OPEN ( ..., IOSTAT=ierr )
   ...
END IF
CALL mp_bcast( ierr, ... , intra_image_comm )
CALL errore( 'routine','error', ierr )
\end{verbatim}
The same applies to all operations performed on a single processor,
or a subgroup of processors: any error code must be broadcast before
the check.
\end{itemize}

\subsection{Data distribution}

Quantum ESPRESSO employ arrays whose memory requirements fall 
into three categories.
\begin{itemize}
\item {\em Fully Scalable}: 
Arrays that are distributed across processors of a pool.
Fully scalable arrays are typically large to very large and contain one 
of the following dimensions:
\begin{itemize}
\item number of plane waves, npw (or max number, npwx)
\item number of Gvectors, ngm
\item number of grid points in the R space, dfft\%nnr
\end{itemize}
Their size decreases linearly with the number of processors in a pool. 

\item {\em Partially Scalable}: 
Arrays that are distributed across processors of the
ortho or diag group. Typically they are much smaller than fully scalable
array, and small in absolute terms for moderate-size system. Their size
however increases quadratically with the number of atoms in the system,
so they have to be distributed for large systems (hundreds to thousands
atoms). Partially scalable arrays contain none of the dimensions listed 
above, two of the following dimensions:
\begin{itemize}
\item number of states, nbnd
\item number of atomic states, natomwfc
\item number of projectors, nkb
\end{itemize}
Their size decreases linearly with the number of processors in a ortho
or diag group. 

\item
{\em Nonscalable}: All the remaining arrays, that are not distributed across
processors. These are typically small arrays, having dimensions like for
instance:
\begin{itemize}
\item number of atoms, nat
\item number of species of atoms, nsp
\end{itemize}
The size of these arrays is independent on the number of processors.
\end{itemize}

% \subsubsection{Parallel fft}

\section{File Formats}

\subsection{Data file(s)}

\qe\ restart file specifications:
Paolo Giannozzi scripsit AD 2005-11-11,
Last modified by Andrea Ferretti 2006-10-29

\subsubsection{Rationale}

Requirements: the data file should be
\begin{itemize}
\item efficient (quick to read and write)
\item easy to read, parse and write without special libraries
\item easy to understand (self-documented)
\item portable across different software packages
\item portable across different computer architectures 
\end{itemize}
Solutions:
\begin{itemize}
\item use binary I/O for large records
\item exploit the file system for organizing data
\item use XML
\item use a small specialized library (iotk) to read, parse, write 
\item ensure the possibility to convert to a portable formatted file
\end{itemize}
Integration with other packages:
\begin{itemize}
\item provide a self-standing (code-independent) library to read/write this format
\item the use of this library is intended to be at high level, hiding low-level details
\end{itemize}

\subsubsection{General structure}

Format name: QEXML \\
Format version: 1.4.0 \\

The "restart file" is actually a "restart directory", containing
several files and sub-directories. For CP/FPMD, the restart directory
is created as "\$prefix\_\$ndw/", where \$prefix is the value of the  
variable "prefix". \$ndw the value of variable ndw, both read in
input; it is read from "\$prefix\_\$ndr/", where \$ndr the value of
variable ndr, read from input. For PWscf, both input and output
directories are called "\$prefix.save/".

The content of the restart directory is as follows:
\begin{verbatim}
data-file.xml          which contains:
                       - general information that doesn't require large data set: 
                         atomic structure, lattice, k-points, symmetries,
                         parameters of the run, ...
                       - pointers to other files or directories containing bulkier
                         data: grids, wavefunctions, charge density, potentials, ...
  
charge_density.dat     contains the charge density
spin_polarization.dat  contains the spin polarization (rhoup-rhodw) (LSDA case)
magnetization.x.dat    
magnetization.y.dat    contain the spin polarization along x,y,z 
magnetization.z.dat    (noncollinear calculations)
lambda.dat             contains occupations (Car-Parrinello dynamics only)
mat_z.1                contains occupations (ensemble-dynamics only)

<pseudopotentials>     A copy of all pseudopotential files given in input
    
<k-point dirs>         Subdirectories K00001/, K00002/, etc, one per k-point.
\end{verbatim}
Each k-point directory contains:
\begin{verbatim}
    evc.dat                wavefunctions for spin-unpolarized calculations, OR
    evc1.dat
    evc2.dat               spin-up and spin-down wavefunctions, respectively, 
                           for spin polarized (LSDA) calculations;
    gkvectors.dat          the details of specific k+G grid;
    eigenval.xml           eigenvalues for the corresponding k-point
                           for spin-unpolarized calculations, OR
    eigenval1.xml          spin-up and spin-down eigenvalues,
    eigenval2.xml          for spin-polarized calculations;
\end{verbatim}
in a molecular dynamics run, also wavefunctions at the preceding time step:
\begin{verbatim}
    evcm.dat               for spin-unpolarized calculations OR
    evcm1.dat
    evcm2.dat              for spin polarized calculations;
\end{verbatim}

\begin{itemize}
\item All files "*.xml" are XML-compliant, formatted file;
\item Files "mat\_z.1", "lambda.dat" are unformatted files, containing a single record;
\item All other files "*.dat", are XML-compliant files, but they
  contain an unformatted record. 
\end{itemize}

\subsubsection{Structure of file "data-file.xml"}

\begin{verbatim}
XML Header: whatever is needed to have a well-formed XML file

Body: introduced by <Root>, terminated by </Root>. Contains first-level tags
      only. These contain only other tags, not values. XML syntax applies.

First-level tags: contain either
     second-level tags, OR
     data tags:   tags containing data (values for a given variable), OR
     file tags:   tags pointing to a file
\end{verbatim}
data tags syntax ( [...] = optional ) :
\begin{verbatim}
      <TAG type="vartype" size="n" [UNIT="units"] [LEN="k"]>
      values (in appropriate units) for variable corresponding to TAG:
      n elements of type vartype (if character, of length k)
      </TAG>
\end{verbatim}
where TAG describes the variable into which data must be read;\\
"vartype" may be "integer", "real", "character", "logical";\\
if type="logical", LEN=k" must be used to specify the length
of the variable character; size="n" is the dimension.\\
Acceptable values for "units" depend on the specific tag.

Short syntax, used only in a few cases:
\begin{verbatim}
      <TAG attribute="something"/> . 
\end{verbatim}
For instance:
\begin{verbatim}
      <FFT_GRID nr1="NR1" nr2="NR2" nr3="NR3"/>
\end{verbatim}
defines the value of the FFT grid parameters nr1, nr2, nr3
for the charge density

\subsubsection{Sample}
Header:
\begin{verbatim}
 <?xml version="1.0"?>
 <?iotk version="1.0.0test"?>
 <?iotk file_version="1.0"?>
 <?iotk binary="F"?> 
\end{verbatim}
These are meant to be used only by iotk (actually they aren't)

First-level tags:
\begin{verbatim}
  - <HEADER>         (global information about fmt version)
  - <CONTROL>        (miscellanea of internal information)
  - <STATUS>         (information about the status of the CP simulation)
  - <CELL>           (lattice vector, unit cell, etc)
  - <IONS>           (type and positions of atoms in the unit cell etc)
  - <SYMMETRIES>     (symmetry operations)
  - <ELECTRIC_FIELD> (details for an eventual applied electric field)
  - <PLANE_WAVES>    (basis set, cutoffs etc)
  - <SPIN>           (info on spin polarizaztion)
  - <MAGNETIZATION_INIT>     (info about starting or constrained magnetization)
  - <EXCHANGE_CORRELATION>
  - <OCCUPATIONS>    (occupancy of the states)
  - <BRILLOUIN_ZONE> (k-points etc)
  - <PARALLELISM>    (specialized info for parallel runs)
  - <CHARGE-DENSITY>
  - <TIMESTEPS>      (positions, velocities, nose' thermostats)
  - <BAND_STRUCTURE_INFO>    (dimensions and basic data about band structure)
  - <EIGENVALUES>    (eigenvalues and related data)
  - <EIGENVECTORS>   (eigenvectors and related data)

  
* Tag description

  <HEADER> 
     <FORMAT>    (name and version of the format)
     <CREATOR>   (name and version of the code generating the file)
  </HEADER>

  <CONTROL>
     <PP_CHECK_FLAG>    (whether file is complete and suitable for post-processing)
     <LKPOINT_DIR>      (whether kpt-data are written in sub-directories)
     <Q_REAL_SPACE>     (whether augmentation terms are used in real space)
     <BETA_REAL_SPACE>  (whether projectors are used in real space, not implemented)
  </CONTROL>

  <STATUS>  (optional, written only by CP)
     <STEP>   (number $n of steps performed, i.e. we are at step $n)
     <TIME>   (total simulation time)
     <TITLE>  (a job descriptor)
     <ekin>   (kinetic energy)
     <eht>    (hartree energy)
     <esr>    (Ewald term, real-space contribution)
     <eself>  (self-interaction of the Gaussians)
     <epseu>  (pseudopotential energy, local)
     <enl>    (pseudopotential energy, nonlocal)
     <exc>    (exchange-correlation energy)
     <vave>   (average of the potential)
     <enthal> (enthalpy: E+PV)
  </STATUS>

  <CELL>
     <NON-PERIODIC_CELL_CORRECTION>
     <BRAVAIS_LATTICE>
     <LATTICE_PARAMETER>
     <CELL_DIMENSIONS>  (cell parameters)
     <DIRECT_LATTICE_VECTORS>
        <UNITS_FOR_DIRECT_LATTICE_VECTORS>
        <a1>
        <a2>
        <a3>
     <RECIPROCAL_LATTICE_VECTORS>
        <UNITS_FOR_RECIPROCAL_LATTICE_VECTORS>
        <b1>
        <b2>
        <b3>
  </CELL>

  <MOVING_CELL> (optional, PW only)
     <CELL_FACTOR>

  <IONS>
     <NUMBER_OF_ATOMS>
     <NUMBER_OF_SPECIES>
     <UNITS_FOR_ATOMIC_MASSES>
     For each $n-th species $X:
        <SPECIE.$n>
           <ATOM_TYPE>
           <MASS>
           <PSEUDO>
        </SPECIE.$n>
     <PSEUDO_DIR>
     <UNITS_FOR_ATOMIC_POSITIONS>
     For each atom $n of species $X:
        <ATOM.$n SPECIES="$X" INDEX=nt tau=(x,y,z) if_pos=...>
  </IONS>

  <SYMMETRIES> (optional, PW only)
     <NUMBER_OF_SYMMETRIES>
     <NUMBER_OF_BRAVAIS_SYMMETRIES>
     <INVERSION_SYMMETRY>
     <DO_NOT_USE_TIME_REVERSAL>
     <TIME_REVERSAL_FLAG>
     <NO_TIME_REV_OPERATIONS>
     <NUMBER_OF_ATOMS>
     <UNITS_FOR_SYMMETRIES>
     For each symmetry $n:
        <SYMM.$n>
           <INFO>
           <ROTATION>
           <FRACTIONAL_TRANSLATION>
           <EQUIVALENT_IONS>
        </SYMM.$n>
     For the remaining bravais symmetries:
        <SYMM.$n>
           <INFO>
           <ROTATION>
        </SYMM.n>
  </SYMMETRIES>

  <ELECTRIC_FIELD>  (optional, sawtooth field in PW only))
     <HAS_ELECTRIC_FIELD> 
     <HAS_DIPOLE_CORRECTION>
     <FIELD_DIRECTION>
     <MAXIMUM_POSITION>
     <INVERSE_REGION>
     <FIELD_AMPLITUDE>
  </ELECTRIC_FIELD>  

  <PLANE_WAVES>
     <UNITS_FOR_CUTOFF>
     <WFC_CUTOFF>
     <RHO_CUTOFF>
     <MAX_NUMBER_OF_GK-VECTORS>
     <GAMMA_ONLY>
     <FFT_GRID>
     <GVECT_NUMBER>
     <SMOOTH_FFT_GRID>
    <SMOOTH_GVECT_NUMBER>
    <G-VECTORS_FILE>       link to file "gvectors.dat"
    <SMALLBOX_FFT_GRID>
  </PLANE_WAVES>

  <SPIN>
     <LSDA>
     <NON-COLINEAR_CALCULATION>
     <SPIN-ORBIT_CALCULATION>
        <SPINOR_DIM>
     <SPIN-ORBIT_DOMAG>
  </SPIN>

  <MAGNETIZATION_INIT>
     <CONSTRAINT_MAG>
     <NUMBER_OF_SPECIES>
     For each species X:
        <SPECIE.$n>
           <STARTING_MAGNETIZATION>
           <ANGLE1>
           <ANGLE2>
           <CONSTRAINT_1,2,3>
        </SPECIE.$n>
        <FIXED_MAGNETIZATION_1,2,3>
        <MAGNETIC_FIELD_1,2,3>
        <TWO_FERMI_ENERGIES>
           <UNITS_FOR_ENERGIES>
           <FIXED_MAGNETIZATION>
           <ELECTRONS_UP>
           <ELECTRONS_DOWN>
           <FERMI_ENERGY_UP>
           <FERMI_ENERGY_DOWN>
        <LAMBDA>
  </MAGNETIZATION_INIT>

  <EXCHANGE_CORRELATION>
     <DFT>
     <LDA_PLUS_U_CALCULATION>
     if LDA_PLUS_U_CALCULATION
        <NUMBER_OF_SPECIES>
        <HUBBARD_LMAX>
        <HUBBARD_L>
        <HUBBARD_U>
        <LDA_PLUS_U_KIND>
        <U_PROJECTION_TYPE>
        <HUBBARD_J>
        <HUBBARD_J0>
        <HUBBARD_ALPHA>
        <HUBBARD_BETA>
     endif
     if <NON_LOCAL_DF>
          <VDW_KERNEL_NAME>
     if <DFT_D2>
          <SCALING_FACTOR>
          <CUTOFF_RADIUS>
     if <XDM>
     if <TKATCHENKO-SCHEFFLER>
          <ISOLATED_SYSTEM>
  </EXCHANGE_CORRELATION>

  if hybrid functional
      <EXACT_EXCHANGE>
        <x_gamma_extrapolation>
        <nqx1>
        <nqx2>
        <nqx3>
        <exxdiv_treatment>
        <yukawa>
        <ecutvcut>
        <exx_fraction>
        <screening_parameter>
      </EXACT_EXCHANGE>
  endif 

  <OCCUPATIONS>
     <SMEARING_METHOD>
     if gaussian smearing
        <SMEARING_TYPE>
        <SMEARING_PARAMETER>
     endif
     <TETRAHEDRON_METHOD>
     if use tetrahedra
        <NUMBER_OF_TETRAHEDRA>
        for each tetrahedron $t
           <TETRAHEDRON.$t>
     endif
     <FIXED_OCCUPATIONS>
     if using fixed occupations
        <INFO>
        <INPUT_OCC_UP>
        if lsda
           <INPUT_OCC_DOWN>
        endif
     endif
  </OCCUPATIONS>

  <BRILLOUIN_ZONE>
     <NUMBER_OF_K-POINTS>
     <UNITS_FOR_K-POINTS>
     <MONKHORST_PACK_GRID>
     <MONKHORST_PACK_OFFSET>
     For each k-point $n:
        <K-POINT.$n>
     <STARTING_F_POINTS>
     For each starting k-point $n:
        <K-POINT_START.$n> kx, ky, kz, wk
     <NORM-OF-Q>
  </BRILLOUIN_ZONE>

  <PARALLELISM>
     <GRANULARITY_OF_K-POINTS_DISTRIBUTION>
     <NUMBER_OF_PROCESSORS>
     <NUMBER_OF_PROCESSORS_PER_POOL>
     <NUMBER_OF_PROCESSORS_PER_IMAGE>
     <NUMBER_OF_PROCESSORS_PER_TASKGROUP>
     <NUMBER_OF_PROCESSORS_PER_POT>
     <NUMBER_OF_PROCESSORS_PER_BAND_GROUP>
     <NUMBER_OF_PROCESSORS_PER_DIAGONALIZATION>
  </PARALLELISM>

  <CHARGE-DENSITY>
      link to file "charge_density.rho"
  </CHARGE-DENSITY>

  <TIMESTEPS>  (optional)
     For each time step $n=0,M
       <STEP$n>
          <ACCUMULATORS>
          <IONS_POSITIONS>
             <stau>
             <svel>
             <taui>
             <cdmi>
             <force>
          <IONS_NOSE>
             <nhpcl>
             <nhpdim>
             <xnhp>
             <vnhp>
          <ekincm>
          <ELECTRONS_NOSE>
             <xnhe>
             <vnhe>
          <CELL_PARAMETERS>
             <ht>
             <htve>
             <gvel>
          <CELL_NOSE>
             <xnhh>
             <vnhh>
          </CELL_NOSE>
  </TIMESTEPS>

  <BAND_STRUCTURE_INFO>
     <NUMBER_OF_BANDS>
     <NUMBER_OF_K-POINTS>
     <NUMBER_OF_SPIN_COMPONENTS>
     <NON-COLINEAR_CALCULATION>
     <NUMBER_OF_ATOMIC_WFC>
     <NUMBER_OF_ELECTRONS>
     <UNITS_FOR_K-POINTS>
     <UNITS_FOR_ENERGIES>
     <FERMI_ENERGY>
  </BAND_STRUCTURE_INFO>

  <EIGENVALUES>
     For all kpoint $n:
         <K-POINT.$n>
             <K-POINT_COORDS>
             <WEIGHT>
             <DATAFILE>                  link to file "./K$n/eigenval.xml"
         </K-POINT.$n>
  </EIGENVALUES>

  <EIGENVECTORS>
     <MAX_NUMBER_OF_GK-VECTORS>
     For all kpoint $n:
         <K-POINT.$n>
             <NUMBER_OF_GK-VECTORS>
             <GK-VECTORS>                link to file "./K$n/gkvectors.dat"
             for all spin $s
                <WFC.$s>                 link to file "./K$n/evc.dat"
                <WFCM.$s>                link to file "./K$n/evcm.dat" (optional)
                                         containing wavefunctions at preceding step
         </K-POINT.n>
  </EIGENVECTORS>
\end{verbatim}

\subsection{Restart files}

\section{Modifying/adding/extending \qe}

\subsection{Programming style (or lack of it)}

There are currently no strict guidelines for developers. You
should however follow at least the following loose ones:
\begin{itemize}
\item Preprocessing options should be capitalized and start with 
two underscores. Examples: \_\_AIX, \_\_LINUX, ...
\item Fortran commands should be capitalized: 
CALL something( )
\item Variable names should be lowercase: \texttt{foo = bar/2}
\item Indent DO's and IF's with three white spaces (editors like emacs will do this automatically for you)
\item Do not write crammed code: leave spaces, insert empty separation lines
\item Use comments (introduced by a !) to explain what is not obvious from 
the code. Remember that what is obvious to you may not be obvious to other 
people. It is especially important to document what a routine does, what
it needs on input, what it produces on output. A few words of comment
may save hours of searching into the code for a piece of missing information.
\item do not use machine-dependent extensions or sloppy syntax. An example:
Standard f90 
requires that a \& is needed both at end of line AND at the beginning of 
continuation line if there is a character variable (inside ' ' or " ")
spanning two lines. Some compilers do not complain if the latter \& is 
missing, others do.
\item use "dp" (defined in module ''kinds'') to define the type of real and complex variables
\item all constants should be defined to be of kind "dp".  Preferred syntax: 0.0\_dp.
\item use "generic" intrinsic functions: SIN, COS, etc.
\item conversions should be explicitely indicated. For conversions to real, 
use DBLE, or else REAL(...,KIND=dp). For conversions to complex, use 
CMPLX(...,...,KIND=dp). For complex conjugate, use CONJG.  For imaginary part, 
use AIMAG.  IMPORTANT: Do not use REAL or CMPLX without KIND=dp, or else you 
will lose precision (except when you take the real part of a 
double precision complex number).
\item Do not use automatic arrays (e.g. \texttt{REAL(dp) :: A(N)} with
\texttt{N} defined at run time) unless you are sure that the array is 
small in all cases: large arrays may easily exceed the stack size,
or the memory size,
\item Do not use pointers unless you have a good reason to: 
pointers may hinder optimization. Allocatable arrays should be used instead.
\item If you use pointers, nullify them before performing tests on their
status.
\item Beware fancy constructs like structures: they look great on paper, 
but they also have the potential to make a code unreadable, or inefficient,
or simply unusable because some compiler gets confused. Avoid nested
structures unless you have a valid reason to use them.
\item Be careful with F90 array syntax and in particular with
array sections. Passing an array section to a routine may look elegant
but it may turn out to be inefficient: a copy will be silently done
if the section is not contiguous in memory (or if the compiler
decides it is the right thing to do), increasing the memory footprint.
\item Do not pass unallocated arrays as arguments, even in those cases where
they are not actually used inside the subroutine: some compilers don't
like it.
\item Do not use any construct that is susceptible to be flagged as 
out-of-bounds error, even if no actual out-of-bound error takes place.
\item Always use IMPLICIT NONE and declare all local variables.
All variables passed as arguments to a routine should be declared as 
INTENT (IN), (OUT), or (INOUT). All variables from modules should be
explicitly specified via USE module, ONLY : variable. Variables used
in an array declaration must be declared first, as in the following
example:
\begin{verbatim}
    INTEGER, INTENT(IN)   :: N 
    REAL(dp), INTENT(OUT) :: A(N) 
\end{verbatim}
in this order (some compilers complain if you put the second line 
before the first).
\end{itemize}

\subsection{Adding or modifying input variables}

New input variables should be added to 
''Modules/input\_parameters.f90'',
then copied to the code internal variables in the ''input.f90''
subroutine. The namelists and cards parsers are in
''Modules/read\_namelists.f90'' and ''Modules/read\_cards.f90''.
Files ''input\_parameters.f90'', ''read\_namelists.f90'',
''read\_cards.f90'' are shared by all codes, while each code
has its own version of ''input.f90''  used to copy input values
into internal variables

EXAMPLE:
suppose you need to add a new input variable called ''pippo''
to the namelist control, then:

\begin{enumerate}
\item add pippo to the input\_parameters.f90 file containing the
namelist control 
\begin{verbatim}
              INTEGER :: pippo = 0
              NAMELIST / control / ....., pippo
\end{verbatim}
Remember: always set an initial value!

\item add pippo to the control\_default subroutine (contained in
module read\_namelists.f90 ) 
\begin{verbatim}
               subroutine control_default( prog )
              ...
              IF( prog == 'PW' ) pippo = 10
              ...
              end subroutine
\end{verbatim}
This routine sets the default value for pippo (can be different in
different codes) 

\item add pippo to the control\_bcast subroutine (contained in module
read\_namelists.f90 ) 
 \begin{verbatim}
                subroutine control_bcast( )
                ...
                call mp_bcast( pippo, intra_image_comm )
                ...
                end subroutine
\end{verbatim}
\end{enumerate}
 
\section{Using SVN}
\label{Sec:SVN}
\qe is maintained in a Subversion (SVN) repository. Developers can have 
read-write access when needed. Note that the latest (development) version
may not work properly, and sometimes not even compile properly. 
Use at your own risk. 

Subversion, also known as SVN, is a software that allows many
developers to work and maintain a single copy of a software in a
central location (repository).
It is installed by default on many Unix machines, or otherwise 
it can be very easily installed.
For the end user, SVN is rather similar to CVS:
if no advanced features are used, the basic commands are the same.
More information on SVN can be found here: 
\texttt{http://subversion.apache.org/}.

Current organization:
\begin{itemize}
\item {\em trunk}: development goes on here -- open read-only to everybody
\item {\em branches}: major new developments, disruptive changes, very
experimental features, things that have a long time before being released
(if ever) ... -- branches may or may not be public
\item {\em external}: packages that are be developed in a separate SVN trunk
can be downloaded into the main QE trunk -- access may be restricted to
specific (usually expert) developers.
\end{itemize}

Follow the instructions in
\texttt{http://qe-forge.org/gf/project/q-e/scmsvn},
under `Access Info'',
to check out (i.e.  download) the SVN repository in either 
read-write or anonymous mode.
The distribution will appear in directory \texttt{trunk/espresso/}.
Branches (i.e. sub-versions) will appear as separate directories.

\subsection{SVN operations}

To update the code to the current version: 
\begin{verbatim}
  svn update
\end{verbatim}
in the directory containing the distribution.
To see the difference between the current version and your modified 
copy:
\begin{verbatim}
  svn diff
\end{verbatim}
To save your modified version into the repository:
(read-write access only):
\begin{verbatim}
  svn commit
\end{verbatim}
Please explain in a few words what your commit is about! Use option
\texttt{-m"comment"} or the editor of your choice (set it using the
\texttt{SVN\_EDITOR} environment variable).
If you want to add a new file, or a new directory, before commiting 
give command
\begin{verbatim}
  svn add
\end{verbatim}
To remove a file/directory (if empty):
\begin{verbatim}
  svn delete
\end{verbatim}
You can move a file (a directory, a group of files, ...) into a different
directory using command
\begin{verbatim}
  svn mv
\end{verbatim}

\subsection{Removing conflicts}
\label{SubSec:Conflicts}
When you update your working copy of the repository, 
you may encounter two types of conflicts:
\begin{enumerate}
\item Somebody else has changed the same lines that you have
      modified. 
\item Somebody else has changed something that has broken one
      or more functionalities of your modified version.
\end{enumerate}
Here we are concerned with kind 1. of conflicts, those that
are noticed by SVN and produce, in addition to a message with
a "C" in the first column before the conflicting file name:
\begin{itemize}
\item \texttt{conflicting-file} containing an attempted merge
of your version with the SVN version, with conflicting sections
indicated by
\begin{verbatim}
   <<<<<<<
     (your version)
   =======
     (SVN version)
   >>>>>>> 
\end{verbatim}
\item \texttt{conflicting-file.mine} containing your version
\item two \texttt{conflicting-file.rXXXXX} containing the two most
recent versions (\texttt{XXXXX} is the revision number) in SVN.
\end{itemize}
Look into the conflicting section(s): in most cases, conflicts are trivial 
(format changes, white spaces) or easily solved (the part of the code you 
were modifying has been moved to another place, or a variable has meanwhilke
changed name, for instance). Edit \texttt{conflicting-file}, remove all other
copies of \texttt{conflicting-file.*}, commit.

Sometimes, the conflict is not so easy to solve. In this case, you
can selectively update your repository at a given date, or at a given
revision number, using command (XXXXX=revision number)
\begin{verbatim}
  svn update -r XXXXX
\end{verbatim}
You can also select a date, using \{"date"\} instead of the revision number.
In this way you can locate which change(s) is (are) the culprit(s).
The web-SVN interface:
\begin{verbatim}
   http://qe-forge.org/gf/project/q-e/scmsvn
\end{verbatim}
will also be very helpful in locating the problem.
Of course, communication with other developers will also help.
The above paragraph applies as well to case 2. os conflicts, in
presence or in absence of explicit SVN conflicts. If the reason for
malfunctioning is not evident, you have to figure out when the
problem started. Once this is done, itis usually straightforward
to figure out why.

\subsection{Merging branch and trunk}
\label{SubSec:Merge}
Let us assume that you have created a branch and that you are working
in the directory of your branch. The simplest way to keep it aligned 
with the trunk is the following command:
\begin{verbatim}
  svn merge ^/trunk/espresso
\end{verbatim}
The caret (\texttt{\^}) syntax is a shorthand for the entire URL of the trunk.
Then you have to remove conflicts that can arise from incompatible changes 
made in the trunk. Then you can commit your "aligned" branch (beware:
the commit message is very large in size if you haven't merged recently;
if so, it may never reach the \texttt{q-e-commits} mailing list).

In order to merge a branch back into the trunk, the simplest procedure is
to align first the branch with the trunk and commit it, as above; then,
in a clean, not locally modified, trunk: 
\begin{verbatim}
  svn merge --reintegrate ^/branches/my-espresso-branch
\end{verbatim}
then, commit.

Note the following very useful property: SVN can merge anything with anything!
The following web page may be useful:
\texttt{http://www.math-linux.com/spip.php?article118}

\subsection{Including a repository into the trunk}
\label{SubSec:propedit}
It is possible to download other repositories into the main \qe\ repository. 
Currently, this possibility works only for GIPAW (and EPW, which
however is not aligned to the svn version); you need to be authorized
to access the GIPAW svn, though. From the trunk/ subdirectory
(the one containing espresso/), type ``svn propedit svn:externals espresso''.
An editor will open. Type the name of the subdirectory of ``espresso/''
where you want the repository to be downloaded, followed by the address of
the repository, exit (not quit!) the editor. Example:
\begin{verbatim}
GIPAW http://qeforge.qe-forge.org/svn/qe-gipaw/trunk
EPW http://qeforge.qe-forge.org/svn/epw-public/trunk/EPW
\end{verbatim}

\section{Bibliography}

Fortran books:
\begin{itemize}
\item 
M. Metcalf, J. Reid, Fortran 95/2003 Explained, Oxford University Press (2004) 
\item
S. J. Chapman, Fortran 95/2003 for Scientists and Engineers, McGraw Hill (2007) 
\item
J. C. Adams, W. S. Brainerd, R. A. Hendrickson, R. E. Maine, J. T. Martin,
B. T. Smith, The Fortran 2003 Handbook, Springer (2009) 
\item
W. S. Brainerd, Guide to Fortran 2003 Programming, Springer (2009)
\end{itemize}
On-line tutorials:
\begin{itemize}
\item Fortran:
http://www.cs.mtu.edu/\~{}shene/COURSES/cs201/NOTES/fortran.html
\item Make:  
http://en.wikipedia.org/wiki/Make\_(software)
\item Configure script:
http://en.wikipedia.org/wiki/Configure\_script
\end{itemize}
(info courtesy of Goranka Bilalbegovic)
\end{document}
